<p>I'm running some containers on Google Container Engine.
One day everything was fine, and the next day I can't <code>attach</code> to my containers anymore. Or <code>exec</code>, or any other docker command.</p>

<p>I deleted the pods and let new ones be instanced, didn't help.
Then I deleted the node and waited for a new one to be created and the pods deployed, didn't help either.</p>

<pre><code>$ kubectl attach www-controller-dev-xxxxx

Error from server: No SSH tunnels currently open. Were the targets able to accept an ssh-key for user "gke-xxxxxxxxxxxxxxxxxxxxxxxx"?
</code></pre>

<p>What else can I try?</p>

<p>The problem might have started after I've deleted the cluster and recreated it, but I can't be sure. Did that before and it never was a problem.</p>

<p>Commands like attach rely on the cluster's master being able to talk to the nodes
in the cluster. However, because the master isn't in the same Compute
Engine network as your cluster's nodes, we rely on SSH tunnels to enable secure
communication.</p>

<p>Container Engine puts an SSH public key in your Compute Engine project
<a href="https://cloud.google.com/compute/docs/metadata" rel="noreferrer">metadata</a>. All Compute Engine VMs using
Google-provided images regularly check their project's common metadata
and their instance's metadata for SSH keys to add to the VM's list of
authorized users. Container Engine also adds a firewall rule to your Compute
Engine network allowing SSH access from the master's IP address to each node
in the cluster.</p>

<p>If kubectl attach (or logs, exec, and port-forward) doesn't work, it's likely that it's because the master is unable to open SSH tunnels to the nodes. To
determine what the underlying problem is, you should check for these potential
causes:</p>

<ol>
<li><p>The cluster doesn't have any nodes.</p>

<p>If you've scaled down the number of nodes in your cluster to zero, SSH
tunnels won't work.</p>

<p>To fix it,
<a href="http://kubernetes.io/docs/user-guide/resizing-a-replication-controller/" rel="noreferrer">resize your cluster</a>
to have at least one node.</p></li>
<li><p>Pods in the cluster have gotten stuck in a terminating state and prevented
nodes that no longer exist from being removed from the cluster.</p>

<p>This is an issue that should only affect Kubernetes version 1.1, but could
be caused by repeated resizing of the cluster down and up.</p>

<p>To fix it,
<a href="http://kubernetes.io/docs/user-guide/kubectl/kubectl_delete/" rel="noreferrer">delete the pods</a>
that have been in a terminating state for more than a few minutes.
The old nodes will then be removed from the master's API and replaced
by the new nodes.</p></li>
<li><p>Your network's firewall rules don't allow for SSH access to the master.</p>

<p>All Compute Engine networks are created with a firewall rule called
"default-allow-ssh" that allows SSH access from all IP addresses (requiring
a valid private key, of course). Container Engine also inserts an SSH rule
for each cluster of the form "gke---ssh"
that allows SSH access specifically from the cluster's master IP to the
cluster's nodes. If neither of these rules exists, then the master will be
unable to open SSH tunnels.</p>

<p>To fix it,
<a href="https://cloud.google.com/compute/docs/networking#addingafirewall" rel="noreferrer">re-add a firewall rule</a>
allowing access to VMs with the tag that's on all the cluster's nodes from
the master's IP address.</p></li>
<li><p>Your project's common metadata entry for sshKeys is full.</p>

<p>If the project's metadata entry named "sshKeys" is close to the 32KiB size
limit, then Container Engine isn't able to add its own SSH key to let it
open SSH tunnels. You can see your project's metadata by running
<code>gcloud compute project-info describe [--project=PROJECT]</code>, then check the
length of the list of sshKeys.</p>

<p>To fix it,
<a href="https://cloud.google.com/compute/docs/instances/adding-removing-ssh-keys#delete_project-wide_ssh_keys" rel="noreferrer">delete some of the SSH keys</a>
that are no longer needed.</p></li>
<li><p>You have set a metadata field with the key "sshKeys" on the VMs in the
cluster.</p>

<p>The node agent on VMs prefers per-instance sshKeys to project-wide SSH keys,
so if you've set any SSH keys specifically on the cluster's nodes, then the
master's SSH key in the project metadata won't be respected by the nodes.
To check, run <code>gcloud compute instances describe &lt;VM-name&gt;</code> and look for
an "sshKeys" field in the metadata.</p>

<p>To fix it,
<a href="https://cloud.google.com/compute/docs/instances/adding-removing-ssh-keys#delete_instance-only_ssh-keys_values" rel="noreferrer">delete the per-instance SSH keys</a>
from the instance metadata.</p></li>
</ol>

<p>It's worth noting that these features are not required for the correct
functioning of the cluster. If you prefer to keep your cluster's network locked
down from all outside access, that's perfectly fine. Just be aware that
features like these won't work as a result.</p>
