<p>I have a Kubernetes cluster that was initialized using the <code>kube-up.sh</code> script inside AWS, and occasionally there's a very slow DNS lookup when finding one service from inside another pod. Here's the basic picture:</p>

<pre><code>    (browser)
        |
        V
      (ELB)
        |
        V
(front-end service)
        |
        V
  (front-end pod)
        |
        V
 (back-end service)
        |
        V
  (back-end pod)
        |
        V
    (database)
</code></pre>

<p>I have timing logging installed at the front-end and back-end levels, and their numbers are wildly divergent for <strong>some requests</strong>. Occasionally we'll see a request that the FE nginx logging says takes 8.3 seconds, but the back-end gunicorn process says takes 30ms.</p>

<p>I can <code>exec</code> into the FE pod and do a <code>curl</code> to the backend endpoint to get timing data according to the example in <a href="https://josephscott.org/archives/2011/10/timing-details-with-curl/" rel="nofollow">this blog post</a>, and it looks like this:</p>

<pre><code>        time_namelookup:  3.513
           time_connect:  3.513
        time_appconnect:  0.000
       time_pretransfer:  3.513
          time_redirect:  0.000
     time_starttransfer:  3.520
                        ----------
             time_total:  3.520
</code></pre>

<p>So the slowness seems to be coming from DNS. We have a separate cluster set up for staging, and this sort of thing doesn't seem to be happening there, so I'm not sure what to make of it. Most requests happen in a reasonable amount of time, less than 50ms, but every tenth one or so takes multiple seconds to resolve.</p>

<p>I found <a href="https://groups.google.com/forum/#!topic/google-containers/oxVw35elJj8" rel="nofollow">this thread</a> that made it sound like SkyDNS's use of etcd might be the problem, but I'm not sure how to verify that or fix it. And this is happening <em>way</em> too often to be periodic missing configuration values (our traffic isn't that high).</p>

<p>There was a bug that was fixed here (<a href="https://github.com/kubernetes/kubernetes/pull/13345" rel="nofollow">https://github.com/kubernetes/kubernetes/pull/13345</a>) that has been shown to cause this problem in Kubernetes clusters 1.0.5 and older.  The problem is fixed in the <a href="https://github.com/kubernetes/kubernetes/releases" rel="nofollow">1.0.6 release</a>.</p>
