title: elastic search nginx proxy timeout
body: <p>"search1" is an AWS elasticsearch service. It has an access policy that only lets traffic through from selected IP addresses. My understanding is that AWS implements this as an ELB in front of a VPC that I cannot access.</p>
  
  <p>"esproxy" is a AWS EC2 instance to act as a proxy to search1. On esproxy, <code>nginx</code> is configured to require (https) basic auth, and anything with that gets proxied to search1. </p>
  
  <p>It works. For a while, hours, or a day. Then every request starts giving "504 Gateway Time-out" errors. <code>nginx</code> still responds instantly to give out 401 auth required errors, but with auth it takes two minutes to get a timeout back. Neither side seems to be under much load when this happens and a restart of <code>nginx</code> fixes it. And really, traffic through the proxy is not heavy, a few thousand hits a day. </p>
  
  <p>Trying to understand the problem, I tried to use <code>openssl</code> like <code>telnet</code>:</p>
  
  <pre><code>openssl s_client -connect search1:443
  [many ssl headers including certs shown rapidly]
  GET / HTTP/1.1
  Host: search1
  
  HTTP/1.1 408 REQUEST_TIMEOUT
  Content-Length:0
  Connection: Close
  </code></pre>
  
  <p>It takes about a minute for that 408 timeout to come back to me. <strong>Aha</strong>, I think, <strong>this particular server is having issuses.</strong> But then I tried that <code>openssl</code> test from another host. Same delay.</p>
  
  <p>Then I think to myself, <strong>hey, <code>curl</code> works to test https, too, now that I know the ssl layer is snappy.</strong> Well, with <code>curl</code> access works, even while <code>nginx</code> and <code>openssl</code> are timing out from the esproxy at that same time.</p>
  
  <p>So I think, <strong>maybe something about the headers? <code>curl</code> has different headers than I'm typing into <code>openssl</code>.</strong> </p>
  
  <p>I modified a low level http/https tool to let me easily send specific headers. And I found it doesn't seem to be lack of or extra headers, but the line endings. <code>nginx</code> (and <code>apache</code>) don't care if you use DOS-style line endings (correct to HTTP spec) or Unix-style (incorrect). The search1 instance (either elastic search itself or the ELB) apparently cares a lot.</p>
  
  <p>Without knowing a whole lot about <code>nginx</code>, I have these questions:</p>
  
  <ol>
  <li>Could the source of my proxy timeouts be a bunch of existing connections caught up with bad request line endings?
  
  <ul>
  <li>How can I tell?</li>
  <li>It might not be since the timeouts are different (one vs two minutes).</li>
  </ul></li>
  <li>Does <code>nginx</code> correct line endings on proxied requests by default?
  
  <ul>
  <li>If not, can it be forced to?</li>
  </ul></li>
  <li><strong>AND if the line endings is a red herring</strong>, how can I get <code>nginx</code> to help me figure this out? All I see in the log is "upstream timed out (110: Connection timed out) while reading response header from upstream", which  doesn't improve my understanding of the issue.</li>
  </ol>
  
  <p>I found this issue earlier in my debugging:
  <a href="https://stackoverflow.com/questions/10395807/nginx-close-upstream-connection-after-request">nginx close upstream connection after request</a>
  And I've already fixed the <code>nginx</code> conf to use a 1.1 proxy as outlined there. Relevant conf:</p>
  
  <pre><code>upstream search1 {
    server search1-abcdefghijklmnopqrstuvwxyz.us-east-1.es.amazonaws.com:443;
  
    # number of connections to keep alive, not time
    keepalive 64;
  }
  
  location / {
    proxy_set_header   X-Forwarded-For $remote_addr;
    proxy_set_header   Host "search1-abcdefghijklmnopqrstuvwxyz.us-east-1.es.amazonaws.com"
    # must supress auth header from being forwarded
    proxy_set_header   Authorization "";
  
    # allow keep-alives on proxied connections
    proxy_http_version 1.1;
    proxy_set_header Connection "";
  
    proxy_pass         https://search1;
  }
  </code></pre>