<p>I have deployed Kubernetes <code>V1.18.2</code> (CDK) using conjure-up (which used bionic)
<em>Update:</em> Destroyed the above env completely and then deployed it back again manually using CDK bundle here <a href="https://jaas.ai/canonical-kubernetes" rel="nofollow noreferrer">https://jaas.ai/canonical-kubernetes</a>, same K8S version same OS version (Ubuntu 18.04) no difference.</p>

<p>The <code>coredns</code> is resolving via <code>/etc/resolv.conf</code>, see <code>configmap</code> below:</p>

<pre><code>Name:         coredns
Namespace:    kube-system
Labels:       cdk-addons=true
Annotations:  
Data
====
Corefile:
----
.:53 {
    errors
    health {
      lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
      fallthrough in-addr.arpa ip6.arpa
    }
    prometheus :9153
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
    loadbalance
}

Events:  &lt;none&gt;
</code></pre>

<p>There is a known issue here at <a href="https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/#known-issues" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/#known-issues</a> about <code>/etc/resolv.conf</code> instead of <code>/run/systemd/resolve/resolv.conf</code></p>

<p>I edited <code>coredns</code> config map to point it to <code>/run/systemd/resolve/resolv.conf</code> but the settings gets reverted.</p>

<p>I also tried setting <code>kubelet-extra-config</code> to <code>{resolvConf: /run/systemd/resolve/resolv.conf}</code>, restarted the server, no changes:</p>

<pre><code>kubelet-extra-config:
    default: '{}'
    description: |
      Extra configuration to be passed to kubelet. Any values specified in this
      config will be merged into a KubeletConfiguration file that is passed to
      the kubelet service via the --config flag. This can be used to override
      values provided by the charm.
      Requires Kubernetes 1.10+.
      The value for this config must be a YAML mapping that can be safely
      merged with a KubeletConfiguration file. For example:
        {evictionHard: {memory.available: 200Mi}}
      For more information about KubeletConfiguration, see upstream docs:
      https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/
    source: user
    type: string
    value: '{resolvConf: /run/systemd/resolve/resolv.conf}'
</code></pre>

<p>But I can see the changes in <code>kubelet</code> config when inspecting the configuration as per <a href="https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/</a></p>

<pre><code>...
"resolvConf": "/run/systemd/resolve/resolv.conf",
...
</code></pre>

<p>This is the error I get in coredns pod:</p>

<pre><code>E0429 09:16:42.172959       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Endpoints: Get https://10.152.183.1:443/api/v1/endpoints?limit=500&amp;resourceVersion=0: dial tcp 10.152.183.1:443: i/o timeout
[INFO] plugin/ready: Still waiting on: "kubernetes"
</code></pre>

<p>See kubernetes service:</p>

<pre><code>default                           kubernetes                               ClusterIP   10.152.183.1     &lt;none&gt;        443/TCP                  4h42m   &lt;none&gt;
</code></pre>

<p>Here is <code>coredns</code> deployment:</p>

<pre><code>Name:                   coredns
Namespace:              kube-system
CreationTimestamp:      Wed, 29 Apr 2020 09:15:07 +0000
Labels:                 cdk-addons=true
                        cdk-restart-on-ca-change=true
                        k8s-app=kube-dns
                        kubernetes.io/name=CoreDNS
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               k8s-app=kube-dns
Replicas:               1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 25% max surge
Pod Template:
  Labels:           k8s-app=kube-dns
  Service Account:  coredns
  Containers:
   coredns:
    Image:       rocks.canonical.com:443/cdk/coredns/coredns-amd64:1.6.7
    Ports:       53/UDP, 53/TCP, 9153/TCP
    Host Ports:  0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  &lt;none&gt;
    Mounts:
      /etc/coredns from config-volume (ro)
  Volumes:
   config-volume:
    Type:               ConfigMap (a volume populated by a ConfigMap)
    Name:               coredns
    Optional:           false
  Priority Class Name:  system-cluster-critical
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    False   ProgressDeadlineExceeded
OldReplicaSets:  &lt;none&gt;
NewReplicaSet:   coredns-6b59b8bd9f (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  11m   deployment-controller  Scaled up replica set coredns-6b59b8bd9f to 1
</code></pre>

<p>Can anyone help, please?</p>

<p><em>More info:</em>
K8S SVC is configured correctly:</p>

<pre><code>Name:              kubernetes
Namespace:         default
Labels:            component=apiserver
                   provider=kubernetes
Annotations:       &lt;none&gt;
Selector:          &lt;none&gt;
Type:              ClusterIP
IP:                10.152.183.1
Port:              https  443/TCP
TargetPort:        6443/TCP
Endpoints:         xx.xx.xx.xx:6443,xx.xx.xx.yy:6443
Session Affinity:  None
Events:            &lt;none&gt;
</code></pre>

<p>I can curl both IP addresses with --insecure</p>

<p>Describing EP:</p>

<pre><code>kubectl describe ep kubernetes 
Name:         kubernetes
Namespace:    default
Labels:       &lt;none&gt;
Annotations:  &lt;none&gt;
Subsets:
  Addresses:          xx.xx.xx.xx,xx.xx.xx.yy
  NotReadyAddresses:  &lt;none&gt;
  Ports:
    Name   Port  Protocol
    ----   ----  --------
    https  6443  TCP

Events:  &lt;none&gt;
</code></pre>

<p>Additional more findings: It looks like most of the <code>vnet</code>s created by juju during <code>CDK</code> deployment are not running. I am suspecting this is because of <code>apparmor</code> (based on <a href="https://jaas.ai/canonical-kubernetes/bundle/21" rel="nofollow noreferrer">https://jaas.ai/canonical-kubernetes/bundle/21</a> )</p>

<blockquote>
  <p>Note: If you desire to deploy this bundle locally on your laptop, see the
  segment about Conjure-Up under Alternate Deployment Methods. Default deployment
  via juju will not properly adjust the apparmor profile to support running
  kubernetes in LXD. At this time, it is a necessary intermediate deployment
  mechanism.</p>
</blockquote>

<pre><code>7: vnet0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel master br1 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:f0:0c:29 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fef0:c29/64 scope link 
       valid_lft forever preferred_lft forever
70: vnet12: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel master br1 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:00:a3:94 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe00:a394/64 scope link 
       valid_lft forever preferred_lft forever
72: vnet13: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel master br1 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:15:17:f4 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe15:17f4/64 scope link 
       valid_lft forever preferred_lft forever
74: vnet14: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel master br1 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:ec:5c:72 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:feec:5c72/64 scope link 
       valid_lft forever preferred_lft forever
76: vnet15: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel master br1 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:60:79:18 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe60:7918/64 scope link 
       valid_lft forever preferred_lft forever
79: vnet16: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel master br1 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:67:ff:14 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe67:ff14/64 scope link 
       valid_lft forever preferred_lft forever
81: vnet17: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel master br1 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:96:71:01 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe96:7101/64 scope link 
       valid_lft forever preferred_lft forever
83: vnet18: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel master br1 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:a8:1d:b7 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fea8:1db7/64 scope link 
       valid_lft forever preferred_lft forever
85: vnet19: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel master br1 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:2a:89:c1 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe2a:89c1/64 scope link 
       valid_lft forever preferred_lft forever
87: vnet20: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel master br1 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:4e:ce:fb brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe4e:cefb/64 scope link 
       valid_lft forever preferred_lft forever
89: vnet21: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel master br1 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:93:55:ac brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe93:55ac/64 scope link 
       valid_lft forever preferred_lft forever
90: vnet22: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc fq_codel master br1 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:b7:ae:b2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:feb7:aeb2/64 scope link 
       valid_lft forever preferred_lft forever
</code></pre>

<p>Another new update:
I tried xenial deployment and noted that <code>/etc/resolv.conf</code> is correctly configured with no issues, however the issue remained the same</p>

<p>It turned out that <code>flannel</code> was conflicting with my local network, specifying the following in the juju's <code>bundle.yaml</code> before deployment:</p>

<pre><code>applications:
  flannel:
    options:
      cidr: 10.2.0.0/16
</code></pre>

<p>Resolved the issue once and for all! :)</p>
