<p>I am using kubernetes 1.15.7 version.</p>

<p>I am trying to follow the link <a href="https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#understanding-the-encryption-at-rest-configuration" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#understanding-the-encryption-at-rest-configuration</a> to enable 'encryption-provider-config' option on 'kube-apiserver'.</p>

<p>I edited file '/etc/kubernetes/manifests/kube-apiserver.yaml' and provided below option</p>

<pre><code>- --encryption-provider-config=/home/rtonukun/secrets.yaml
</code></pre>

<p>But after that I am getting below error.</p>

<pre><code>The connection to the server 171.69.225.87:6443 was refused - did you specify the right host or port?
</code></pre>

<p>with all kubectl commands like 'kubectl get no'.</p>

<p>Mainy, how do I do these below two steps?</p>

<pre><code>3. Set the --encryption-provider-config flag on the kube-apiserver to point to the location of the config file.

4. Restart your API server.
</code></pre>

<p>I've reproduced exactly your scenario, and I'll try to explain how I fixed it</p>
<h3>Reproducing the same scenario</h3>
<ol>
<li>Create the encrypt file on <code>/home/koopakiller/secrets.yaml</code>:</li>
</ol>
<pre><code>apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    providers:
    - aescbc:
        keys:
        - name: key1
          secret: r48bixfj02BvhhnVktmJJiuxmQZp6c0R60ZQBFE7558=
    - identity: {}
</code></pre>
<p>Edit the file <code>/etc/kubernetes/manifests/kube-apiserver.yaml</code> and set the <code>--encryption-provider-config</code> flag:</p>
<pre><code>   - --encryption-provider-config=/home/koopakiller/encryption.yaml
</code></pre>
<p>Save the file and exit.</p>
<p>When I checked the pods status got the same error:</p>
<pre><code>$ kubectl get pods -A
The connection to the server 10.128.0.62:6443 was refused - did you specify the right host or port?
</code></pre>
<h3>Troubleshooting</h3>
<p>Since kubectl is not working anymore, I tried to look directly the running containers using docker command, then I see kube-apiserver container was recently recreated:</p>
<pre><code>$ docker ps 
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS               NAMES
54203ea95e39        k8s.gcr.io/pause:3.1   &quot;/pause&quot;                 1 minutes ago       Up 1 minutes                            k8s_POD_kube-apiserver-lab-1_kube-system_015d9709c9881516d6ecf861945f6a10_0
...
</code></pre>
<p>Kubernetes store the logs of created pods on <code>/var/log/pods</code> directory, I've checked the kube-apiserver log file and found a valuable information:</p>
<blockquote>
<p>{&quot;log&quot;:&quot;Error: error opening encryption provider configuration file &quot;/home/koopakiller/encryption.yaml&quot;: open <strong>/home/koopakiller/encryption.yaml: no such file or directory</strong>\n&quot;,&quot;stream&quot;:&quot;stderr&quot;,&quot;time&quot;:&quot;2020-01-22T13:28:46.772768108Z&quot;}</p>
</blockquote>
<h3>Explanation</h3>
<p>Taking a look at manifest file <code>kube-apiserver.yaml</code> is possible to see the command <code>kube-apiserver</code>, it runs into container, so they need to have the <code>encryption.yaml</code> file mounted into container.</p>
<p>If you check the <code>volumeMounts</code> in this file, you could see that only the paths below is mounted in container by default:</p>
<blockquote>
<ul>
<li>/etc/ssl/certs</li>
<li>/etc/ca-certificates</li>
<li>/etc/kubernetes/pki</li>
<li>/usr/local/share/ca-certificates</li>
<li>/usr/share/ca-certificates</li>
</ul>
</blockquote>
<pre><code>
    ...
        volumeMounts:
        - mountPath: /etc/ssl/certs
          name: ca-certs
          readOnly: true
        - mountPath: /etc/ca-certificates
          name: etc-ca-certificates
          readOnly: true
        - mountPath: /etc/kubernetes/pki
          name: k8s-certs
          readOnly: true
        - mountPath: /usr/local/share/ca-certificates
          name: usr-local-share-ca-certificates
          readOnly: true
        - mountPath: /usr/share/ca-certificates
          name: usr-share-ca-certificates
          readOnly: true
    ...
</code></pre>
<p>Based on the facts above, we can assume that apiserver failed to start because <code>/home/koopakiller/encryption.yaml</code> doesn't actually mounted into container.</p>
<h3>How to solve</h3>
<p>I can see 2 ways to solve this issue:</p>
<p><strong>1st</strong> - Copy the encryption file to <code>/etc/kubernetes/pki</code> (or any of the path above) and change the path in <code>/etc/kubernetes/kube-apiserver.yaml</code>:</p>
<pre><code>   - --encryption-provider-config=/etc/kubernetes/encryption.yaml
</code></pre>
<p>Save the file and wait apiserver restart.</p>
<p><strong>2nd</strong> - Create a new <code>volumeMounts</code> in the kube-apiserver.yaml manifest to mount a custom directory from node into container.</p>
<p>Let's create a new directory in <code>/etc/kubernetes/secret</code> (home folder isn't a good location to leave config files =)).</p>
<p>Edit <code>/etc/kubernetes/manifests/kube-apiserver.yaml</code>:</p>
<pre><code>...
    - --encryption-provider-config=/etc/kubernetes/secret/encryption.yaml
...
  volumeMounts:
    - mountPath: /etc/kubernetes/secret
      name: secret
      readOnly: true
...
  volumes:
    - hostPath:
      path: /etc/kubernetes/secret
      type: DirectoryOrCreate
    name: secret
...
</code></pre>
<p>After save the file kubernetes will mount the node path <code>/etc/kubernetes/secret</code> into the same path into the apiserver container, wait start completely and try to list your node again.</p>
<p>Please let know if that helped!</p>
