title: intermittent nginx timeouts having jetty as upstream server
body: <p>We do have a weird production issue of <code>nginx</code> timeouts. The timeout says below :</p>
  
  <pre><code>"upstream timed out(110: Connection timed out)" 
  </code></pre>
  
  <p>Here, the upstream server is <code>jetty</code> running on the same host where nginx is also running. Jetty is running on port <code>8080</code> where nginx is running on <code>443</code>. </p>
  
  <p>Having checked the above error, I verified the logs of jetty logs and nginx logs. Though jetty is returning the response in less than a second, the response sent to nginx is being <code>delayed</code> by close to <code>60 secs</code>. And nginx is timing is firing the timedout alert after 60 secs with http response code of <code>504</code>. This is happening <code>intermittently</code>. </p>
  
  <p>Below are the logs of jetty &amp; nginx for a timed out request. </p>
  
  <p><strong>Jetty log :</strong></p>
  
  <blockquote>
    <p>{"evt":1494519426927,"intelId":"50","intelSeq":112506,"intelVer":"1","time":"2017-05-11T16:17:06.927Z","uiCorrelationIdV1":"SUI-1494519425839-42047","threadName":"qtp754853679-357","wResource":"<a href="http://m.xxx.com/search/facet/women/womens-handbags-9780510203/shoulder/_/N-53f3Z7hk9Z1z0roil" rel="noreferrer">http://m.xxx.com/search/facet/women/womens-handbags-9780510203/shoulder/_/N-53f3Z7hk9Z1z0roil</a>","wMethod":"GET","wStatus":200,"wDurationMicros":1087801,"wJlpLocation":"","wFwdFor":"<em>NADA</em>","wHostHdr":"m.xxxx.com","wReferer":"<a href="https://m.xxxx.com/search/facet/women/womens-handbags-9780510203/_/N-53f3Z7hk9?search-term=Bags&amp;sortBy=priceLow&amp;facet=Handbag%20Style" rel="noreferrer">https://m.xxxx.com/search/facet/women/womens-handbags-9780510203/_/N-53f3Z7hk9?search-term=Bags&amp;sortBy=priceLow&amp;facet=Handbag%20Style</a>","wHttpVer":"HTTP/1.0","wWsgClientIp":"80.192.191.2","wSrcIP":"127.0.0.1","wUserAgent":"Mozilla/5.0
    (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) AppleWebKit/603.1.30
    (KHTML, like Gecko) Version/10.0 Mobile/14E304
    Safari/602.1","intelCropped":false,"intelLength":834}</p>
  </blockquote>
  
  <p><strong>Nginx log :</strong></p>
  
  <blockquote>
    <p>2017-05-11T17:18:09+01:00 intelId="56" intelVer="2" wMethod="GET"
    wResource="/search/facet/women/womens-handbags-9780510203/across-body/shoulder/<em>/N-53f3Z7hk9Z1z0roq4Z1z0roil?search-term=Bags&amp;sortBy=priceLow&amp;facet=Handbag%20Style"
    wStatus="504" wCacheStatus="MISS" wSrcIP="172.17.233.135" wSize="176"
    wDurationSeconds="60.001" wHostHdr="m.xxx.com"
    wReferer="https://m.xxxx.com/search/facet/women/womens-handbags-9780510203/shoulder/</em>/N-53f3Z7hk9Z1z0roil?search-term=Bags&amp;sortBy=priceLow&amp;facet=Handbag%20Style"
    wSSL="on" wSSLver="771" wSSLciph="TLS1.2-ECDHE-RSA-AES256-GCM-SHA384"
    wWsgClientIp="80.192.191.2" wFwdFor="-" wJlpLocation="-"
    wProtocol="HTTP/1.1" wUpstreamAddr="127.0.0.1:8080" wPort=443
    s_vi="[CS]v1|2B4D4CA80501261B-600001064000B11D[CE]"
    s_ppv="jl%253Asearch%2C14%2C100%2C5789%2C414%2C628%2C414%2C736%2C3%2CP" recognisedUser="true" wUiCorrelationIdV1="-" wUserAgent="Mozilla/5.0
    (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) AppleWebKit/603.1.30
    (KHTML, like Gecko) Version/10.0 Mobile/14E304 Safari/602.1"
    deviceType="mobile"</p>
  </blockquote>
  
  <p>From the above two logs, we can infer that jetty returned a 200 response at <code>2017-05-11T16:17:06.927Z</code>, but it was received by nginx at <code>2017-05-11T17:18:09+01:00</code>. This 60 plus secs is causing the timeouts. Its bit weird as both nginx and jetty are hosted on the same host. </p>
  
  <p>If someone could help us in debugging the issue or provide suggestions, it would be great. </p>
  
  <p>Many Thanks in advance. </p>