title: 'Unable to connect Net/http: TLS handshake timeout' — Why can't Kubectl connect to Azure Kubernetes server? (AKS)
body: <blockquote>
    <p>My question (to MS and anyone else) is: Why is this issue occurring and what work around can be implemented by the users / customers themselves as opposed to by Microsoft Support?</p>
  </blockquote>
  
  <p>There have obviously been 'a few' other question about this issue:</p>
  
  <ol>
  <li><a href="https://stackoverflow.com/questions/48229824/managed-azure-kubernetes-connection-error">Managed Azure Kubernetes connection error</a></li>
  <li><a href="https://stackoverflow.com/questions/48761952/cant-contact-our-azure-aks-kube-tls-handshake-timeout">Can&#39;t contact our Azure-AKS kube - TLS handshake timeout</a></li>
  <li><a href="https://stackoverflow.com/questions/47026265/azure-kubernetes-tls-handshake-timeout">Azure Kubernetes: TLS handshake timeout</a> (this one has some Microsoft feedback)</li>
  </ol>
  
  <p>And multiple GitHub issues posted to the AKS repo:</p>
  
  <ol>
  <li><a href="https://github.com/Azure/AKS/issues/112" rel="noreferrer">https://github.com/Azure/AKS/issues/112</a></li>
  <li><a href="https://github.com/Azure/AKS/issues/124" rel="noreferrer">https://github.com/Azure/AKS/issues/124</a></li>
  <li><a href="https://github.com/Azure/AKS/issues/164" rel="noreferrer">https://github.com/Azure/AKS/issues/164</a></li>
  <li><a href="https://github.com/Azure/AKS/issues/177" rel="noreferrer">https://github.com/Azure/AKS/issues/177</a></li>
  <li><a href="https://github.com/Azure/AKS/issues/324" rel="noreferrer">https://github.com/Azure/AKS/issues/324</a></li>
  </ol>
  
  <p>Plus a few twitter threads:</p>
  
  <ol>
  <li><a href="https://twitter.com/ternel/status/955871839305261057" rel="noreferrer">https://twitter.com/ternel/status/955871839305261057</a></li>
  </ol>
  
  <h2>TL;DR</h2>
  
  <blockquote>
    <p><em><a href="https://stackoverflow.com/questions/50726534/unable-to-connect-net-http-tls-handshake-timeout-why-cant-kubectl-connect?answertab=votes#tab-top">Skip to workarounds in Answers below</a></em>.</p>
  </blockquote>
  
  <p>Current best solution is post a help ticket — and wait — or re-create your AKS cluster (maybe more than once, cross your fingers, see below...) but there should be something better. <strong>At least please grant the ability to let AKS preview customers, regardless of support tier, upgrade their support request severity for THIS specific issue.</strong></p>
  
  <p>You can also try scaling your Cluster (assuming that doesn't break your app).</p>
  
  <h2>What about GitHub?</h2>
  
  <p>Many of the above GitHub issues have been closed as resolved but the issue persists.  Previously there was an announcements document regarding the problem but no such status updates are currently available even though the problem continues to present itself:</p>
  
  <ol>
  <li><a href="https://github.com/Azure/AKS/tree/master/annoucements" rel="noreferrer">https://github.com/Azure/AKS/tree/master/annoucements</a></li>
  </ol>
  
  <p>I am posting this as I have a few new tidbits that I haven't seen elsewhere and I am wondering if anyone has ideas as far as other potential options for working around the issue.</p>
  
  <h2>Affected VM / Node Resource Usage</h2>
  
  <p>The first piece I haven't seen mentioned elsewhere is Resource usage on the nodes / vms / instances that are being impacted by the above Kubectl 'Unable to connect to the server: net/http: TLS handshake timeout' issue.  </p>
  
  <h3>Production Node Utilization</h3>
  
  <p>The node(s) on my impacted cluster look like this:</p>
  
  <p><a href="https://i.stack.imgur.com/i9Oc4.png" rel="noreferrer"><img src="https://i.stack.imgur.com/i9Oc4.png" alt="net/http: TLS handshake timeout"></a></p>
  
  <p>The drop in utilization and network io correlates strongly with both the increase in disk utilization AND the time period we began experiencing the issue. </p>
  
  <p>The overall Node / VM utilization is generally flat prior to this chart for the previous 30 days with a few bumps relating to production site traffic / update pushes etc.</p>
  
  <h2>Metrics After Issue Mitigation <strong><em>(Added Postmortem)</em></strong></h2>
  
  <p>To the above point, here are the metrics the same Node after Scaling up and then back down (which happened to alleviate our issue, but does not always work — see answers at bottom):</p>
  
  <p><a href="https://i.stack.imgur.com/LVma8.png" rel="noreferrer"><img src="https://i.stack.imgur.com/LVma8.png" alt="enter image description here"></a></p>
  
  <p><strong><em>Notice the 'Dip' in CPU and Network?</em></strong>  That's where the Net/http: TLS issue impacted us — and when the AKS Server was un-reachable from Kubectl.  Seems like it wasn't talking to the VM / Node in addition to not responding to our requests.</p>
  
  <p>As soon as we were back (scaled the # nodes up by one, and back down — see answers for workaround) the Metrics (CPU etc) went back to normal — and we could connect from Kubectl.  This means we can probably create an Alarm off of this behavior (and I have a issue in asking about this on Azure DevOps side: <a href="https://github.com/Azure/AKS/issues/416" rel="noreferrer">https://github.com/Azure/AKS/issues/416</a>)</p>
  
  <h1>Node Size Potentially Impacts Issue Frequency</h1>
  
  <p>Zimmergren over on GitHub indicates that he has less issues with larger instances than he did running bare bones smaller nodes.  This makes sense to me and could indicate that the way the AKS servers divy up the workload (see next section) could be based on the size of the instances.</p>
  
  <p>"The size of the nodes (e.g. D2, A4, etc) :)
  I've experienced that when running A4 and up, my cluster is healther than if running A2, for example. (And I've got more than a dozen similar experiences with size combinations and cluster failures, unfortunately)." (<a href="https://github.com/Azure/AKS/issues/268#issuecomment-375715435" rel="noreferrer">https://github.com/Azure/AKS/issues/268#issuecomment-375715435</a>)</p>
  
  <p>Other Cluster size impact references:</p>
  
  <ol>
  <li>giorgited (<a href="https://github.com/Azure/AKS/issues/268#issuecomment-376390692" rel="noreferrer">https://github.com/Azure/AKS/issues/268#issuecomment-376390692</a>)</li>
  </ol>
  
  <blockquote>
    <p><em>An AKS server responsible for more smaller Clusters may possibly get hit more often?</em></p>
  </blockquote>
  
  <h2>Existence of Multiple AKS Management 'Servers' in one Az Region</h2>
  
  <p>The next thing I haven't seen mentioned elsewhere is the fact that you can have multiple Clusters running side by side in the same Region where one Cluster (production for us in this case) gets hit with 'net/http: TLS handshake timeout' and the other is working fine and can be connected to normally via Kubectl (for us this is our identical staging environment).</p>
  
  <p>The fact that users (Zimmergren etc above) seem to feel that the Node size impacts the likelihood that this issue will impact you also seems to indicate that node size may relate to the way the sub-region responsibilities are assigned to the sub-regional AKS management servers. </p>
  
  <blockquote>
    <p>That could mean that re-creating your cluster with a different Cluster size would be more likely to place you on a different management server — alleviating the issue and reducing the likelihood that multiple re-creations would be necessary.</p>
  </blockquote>
  
  <h3>Staging Cluster Utilization</h3>
  
  <p>Both of our AKS Clusters are in U.S. East.  As a reference to the above 'Production' Cluster metrics our 'Staging' Cluster (also U.S. East) resource utilization does not have the massive drop in CPU / Network IO — AND does not have the increase in disk etc. over the same period:</p>
  
  <p><a href="https://i.stack.imgur.com/hL5Ux.png" rel="noreferrer"><img src="https://i.stack.imgur.com/hL5Ux.png" alt="net/http: TLS handshake timeout staging instance is reachable via kubectl."></a></p>
  
  <h3>Identical Environments are Impacted Differently</h3>
  
  <p>Both of our Clusters are running identical ingresses, services, pods, containers so it is also unlikely that anything a user is doing causes this problem to crop up.</p>
  
  <h3>Re-creation is only SOMETIMES successful</h3>
  
  <p>The above existence of multiple AKS management server sub-regional responsibilities makes sense with the behavior described by other users on github (<a href="https://github.com/Azure/AKS/issues/112" rel="noreferrer">https://github.com/Azure/AKS/issues/112</a>) where some users are able to re-create a cluster (which can then be contacted) while others re-create and still have issues.  </p>
  
  <h3>Emergency could = Multiple Re-Creations</h3>
  
  <p>In an emergency (ie your production site... like ours... needs to be managed) you can <strong><em>PROBABLY</em></strong> just re-create until you get a working cluster that happens to land on a different AKS management server instance (one that is not impacted) but be aware that this may not happen on your first attempt — AKS cluster re-creation is not exactly instant.</p>
  
  <p>That said...</p>
  
  <h1>Resources on the Impacted Nodes Continue to Function</h1>
  
  <p>All of the containers / ingresses / resources on our impacted VM appear to be working well and I don't have any alarms going off for up-time / resource monitoring (other than the utilization weirdness listed above in the graphs)</p>
  
  <blockquote>
    <p>I want to know why this issue is occurring and what work around can be implemented by the users themselves as opposed to by Microsoft Support (currently have a ticket in). If you have an idea let me know.</p>
  </blockquote>
  
  <h2>Potential Hints at the Cause</h2>
  
  <ol>
  <li><a href="https://github.com/Azure/AKS/issues/164#issuecomment-363613110" rel="noreferrer">https://github.com/Azure/AKS/issues/164#issuecomment-363613110</a></li>
  <li><a href="https://github.com/Azure/AKS/issues/164#issuecomment-365389154" rel="noreferrer">https://github.com/Azure/AKS/issues/164#issuecomment-365389154</a></li>
  </ol>
  
  <h2>Why no GKE?</h2>
  
  <p>I understand that Azure AKS is in preview and that a lot of people have moved to GKE because of this problem ().  That said my Azure experience has been nothing but positive thus far and I would prefer to contribute a solution if at all possible.</p>
  
  <p>And also... GKE occasionally faces something similar:</p>
  
  <ol>
  <li><a href="https://stackoverflow.com/questions/47481022/tls-handshake-timeout-with-kubernetes-in-gke?rq=1">TLS handshake timeout with kubernetes in GKE</a></li>
  </ol>
  
  <p>I would be interested to see if scaling the nodes on GKE also solved the problem over there.</p>