<p>I am creating a cluster of 1 master 2 nodes kubernetes. I am trying to create the skydns based on the following:</p>

<pre><code>apiVersion: v1
kind: ReplicationController
metadata:
  name: kube-dns-v11
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    version: v11
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 1
  selector:
    k8s-app: kube-dns
    version: v11
  template:
    metadata:
      labels:
        k8s-app: kube-dns
        version: v11
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: etcd
        image: gcr.io/google_containers/etcd-amd64:2.2.1
        # resources:
        #   # TODO: Set memory limits when we've profiled the container for large
        #   # clusters, then set request = limit to keep this container in
        #   # guaranteed class. Currently, this container falls into the
        #   # "burstable" category so the kubelet doesn't backoff from restarting it.
        #   limits:
        #     cpu: 100m
        #     memory: 500Mi
        #   requests:
        #     cpu: 100m
        #     memory: 50Mi
        command:
        - /usr/local/bin/etcd
        - -data-dir
        - /var/etcd/data
        - -listen-client-urls
        - http://127.0.0.1:2379,http://127.0.0.1:4001
        - -advertise-client-urls
        - http://127.0.0.1:2379,http://127.0.0.1:4001
        - -initial-cluster-token
        - skydns-etcd
        volumeMounts:
        - name: etcd-storage
          mountPath: /var/etcd/data
      - name: kube2sky
        image: gcr.io/google_containers/kube2sky:1.14
        # resources:
        #   # TODO: Set memory limits when we've profiled the container for large
        #   # clusters, then set request = limit to keep this container in
        #   # guaranteed class. Currently, this container falls into the
        #   # "burstable" category so the kubelet doesn't backoff from restarting it.
        #   limits:
        #     cpu: 100m
        #     # Kube2sky watches all pods.
        #     memory: 200Mi
        #   requests:
        #     cpu: 100m
        #     memory: 50Mi
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          # successThreshold: 1
          # failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /readiness
            port: 8081
            scheme: HTTP
          # we poll on pod startup for the Kubernetes master service and
          # only setup the /readiness HTTP server once that's available.
          initialDelaySeconds: 30
          timeoutSeconds: 5
        args:
        # command = "/kube2sky"
        - --domain=cluster.local
      - name: skydns
        image: gcr.io/google_containers/skydns:2015-10-13-8c72f8c
        resources:
          # TODO: Set memory limits when we've profiled the container for large
          # clusters, then set request = limit to keep this container in
          # guaranteed class. Currently, this container falls into the
          # "burstable" category so the kubelet doesn't backoff from restarting it.
          limits:
            cpu: 100m
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 50Mi
        args:
        # command = "/skydns"
        - -machines=http://127.0.0.1:4001
        - -addr=0.0.0.0:53
        - -ns-rotate=false
        - -domain=cluster.local
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
      - name: healthz
        image: gcr.io/google_containers/exechealthz:1.0
        # resources:
        #   # keep request = limit to keep this container in guaranteed class
        #   limits:
        #     cpu: 10m
        #     memory: 20Mi
        #   requests:
        #     cpu: 10m
        #     memory: 20Mi
        args:
        - -cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1 &gt;/dev/null
        - -port=8080
        ports:
        - containerPort: 8080
          protocol: TCP
      volumes:
      - name: etcd-storage
        emptyDir: {}
      dnsPolicy: Default  # Don't use cluster DNS.
</code></pre>

<p>However, skydns is spitting out the following:</p>

<pre><code>&gt; $kubectl logs kube-dns-v11-k07j9 --namespace=kube-system  skydns 
&gt; 2016/04/18 12:47:05 skydns: falling back to default configuration,
&gt; could not read from etcd: 100: Key not found (/skydns) [1]  2016/04/18
&gt; 12:47:05 skydns: ready for queries on cluster.local. for
&gt; tcp://0.0.0.0:53 [rcache 0]  2016/04/18 12:47:05 skydns: ready for
&gt; queries on cluster.local. for udp://0.0.0.0:53 [rcache 0]  2016/04/18
&gt; 12:47:11 skydns: failure to forward request "read udp
&gt;     192.168.122.1:53: i/o timeout" 2016/04/18 12:47:15 skydns: failure to forward request "read udp 192.168.122.1:53: i/o timeout" 2016/04/18
&gt; 12:47:19 skydns: failure to forward request "read udp
&gt;     192.168.122.1:53: i/o timeout" 2016/04/18 12:47:23 skydns: failure to forward request "read udp 192.168.122.1:53: i/o timeout" 2016/04/18
&gt; 12:47:27 skydns: failure to forward request "read udp
&gt;     192.168.122.1:53: i/o timeout" 2016/04/18 12:47:31 skydns: failure to forward request "read udp 192.168.122.1:53: i/o timeout" 2016/04/18
&gt; 12:47:35 skydns: failure to forward request "read udp
&gt;     192.168.122.1:53: i/o timeout" 2016/04/18 12:47:39 skydns: failure to forward request "read udp 192.168.122.1:53: i/o timeout" 2016/04/18
&gt; 12:47:43 skydns: failure to forward request "read udp
&gt;     192.168.122.1:53: i/o timeout" 2016/04/18 12:47:47 skydns: failure to forward request "read udp 192.168.122.1:53: i/o timeout" 2016/04/18
&gt; 12:47:51 skydns: failure to forward request "read udp
&gt;     192.168.122.1:53: i/o timeout" 2016/04/18 12:47:55 skydns: failure to forward request "read udp 192.168.122.1:53: i/o timeout" 2016/04/18
&gt; 12:47:59 skydns: failure to forward request "read udp
&gt;     192.168.122.1:53: i/o timeout" 2016/04/18 12:48:03 skydns: failure to forward request "read udp 192.168.122.1:53: i/o timeout"
</code></pre>

<p>After looking further, I just realized what is a 192.168.122.1? It is virtual switch on kvm. Why is skydns trying to hit my virtual switch or dns server of virtual machine?</p>

<p><code>SkyDNS</code> defaults its forwarding nameservers to the one listed in <code>/etc/resolv.conf</code>. Since <code>SkyDNS</code> runs inside the <code>kube-dns</code> pod as a cluster addon, it inherits its <code>/etc/resolv.conf</code> from its host as described in the <a href="https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns#inheriting-dns-from-the-node" rel="noreferrer"><code>kube-dns</code> doc</a>. </p>

<p>From your question, it looks like your host's <code>/etc/resolv.conf</code> is configured to use 192.168.122.1 as its nameserver and hence that becomes the forwarding server in your <code>SkyDNS</code> config. I believe 192.168.122.1 is not routable from your Kubernetes cluster and that's why you are seeing "failure to forward request" errors in the <code>kube-dns</code> logs.</p>

<p>The simplest solution to this problem is to supply a reachable DNS server as a flag to <code>SkyDNS</code> in your RC config. Here is an example (it's just your RC config, but adds the <code>-nameservers</code> flag in the <code>SkyDNS</code> container spec):</p>

<pre><code>apiVersion: v1
kind: ReplicationController
metadata:
  name: kube-dns-v11
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    version: v11
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 1
  selector:
    k8s-app: kube-dns
    version: v11
  template:
    metadata:
      labels:
        k8s-app: kube-dns
        version: v11
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: etcd
        image: gcr.io/google_containers/etcd-amd64:2.2.1
        # resources:
        #   # TODO: Set memory limits when we've profiled the container for large
        #   # clusters, then set request = limit to keep this container in
        #   # guaranteed class. Currently, this container falls into the
        #   # "burstable" category so the kubelet doesn't backoff from restarting it.
        #   limits:
        #     cpu: 100m
        #     memory: 500Mi
        #   requests:
        #     cpu: 100m
        #     memory: 50Mi
        command:
        - /usr/local/bin/etcd
        - -data-dir
        - /var/etcd/data
        - -listen-client-urls
        - http://127.0.0.1:2379,http://127.0.0.1:4001
        - -advertise-client-urls
        - http://127.0.0.1:2379,http://127.0.0.1:4001
        - -initial-cluster-token
        - skydns-etcd
        volumeMounts:
        - name: etcd-storage
          mountPath: /var/etcd/data
      - name: kube2sky
        image: gcr.io/google_containers/kube2sky:1.14
        # resources:
        #   # TODO: Set memory limits when we've profiled the container for large
        #   # clusters, then set request = limit to keep this container in
        #   # guaranteed class. Currently, this container falls into the
        #   # "burstable" category so the kubelet doesn't backoff from restarting it.
        #   limits:
        #     cpu: 100m
        #     # Kube2sky watches all pods.
        #     memory: 200Mi
        #   requests:
        #     cpu: 100m
        #     memory: 50Mi
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          # successThreshold: 1
          # failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /readiness
            port: 8081
            scheme: HTTP
          # we poll on pod startup for the Kubernetes master service and
          # only setup the /readiness HTTP server once that's available.
          initialDelaySeconds: 30
          timeoutSeconds: 5
        args:
        # command = "/kube2sky"
        - --domain=cluster.local
      - name: skydns
        image: gcr.io/google_containers/skydns:2015-10-13-8c72f8c
        resources:
          # TODO: Set memory limits when we've profiled the container for large
          # clusters, then set request = limit to keep this container in
          # guaranteed class. Currently, this container falls into the
          # "burstable" category so the kubelet doesn't backoff from restarting it.
          limits:
            cpu: 100m
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 50Mi
        args:
        # command = "/skydns"
        - -machines=http://127.0.0.1:4001
        - -addr=0.0.0.0:53
        - -ns-rotate=false
        - -domain=cluster.local
        - -nameservers=8.8.8.8:53,8.8.4.4:53  # Adding this flag. Dont use double quotes.
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
      - name: healthz
        image: gcr.io/google_containers/exechealthz:1.0
        # resources:
        #   # keep request = limit to keep this container in guaranteed class
        #   limits:
        #     cpu: 10m
        #     memory: 20Mi
        #   requests:
        #     cpu: 10m
        #     memory: 20Mi
        args:
        - -cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1 &gt;/dev/null
        - -port=8080
        ports:
        - containerPort: 8080
          protocol: TCP
      volumes:
      - name: etcd-storage
        emptyDir: {}
      dnsPolicy: Default  # Don't use cluster DNS.
</code></pre>
