title: Nginx reverse proxy is very slow
body: <p>i'm having problems with nginx as reverse proxy to tornado (python server) instances.</p>
  <p>We have 3 servers (dns round robin) with the same configuration, and the problem only exists on one, several times each day. It seems random.</p>
  <p>The problem is a very slow response times (2,3s to 2minutes!) on proxy pass requests. Most of requests get an answer under one second.</p>
  <p>We have a cron job that 'pings' this server every minute. ICMP pings are always OK, static resources get are always OK, but we have 5-50/day requests to tornado via nginx that are really slow.
  The nginx is in front of 8 instances of tornado servers with a mongo DB. Each nginx server gets 100-200 requests/minute.</p>
  <p>When looking at the app logs, the response times on tornado side are never above one second, the problem really seems to be from nginx-tornado interface.</p>
  <p>System monitoring (disk, memory, cpu) is always OK and &lt;30% use. The server houses nginx, tornado and mongodb and is 32gb RAM with 8 threads. Under Ubuntu 18LTS.</p>
  <p>I will post the nginx config i need to anonymize, we only have a site-available config that is not crazy.</p>
  <p>How can i go further on diagnostic ? I think there is a stacking/enqueue of requests somewhere.</p>
  <p>some info:</p>
  <pre><code># ss -lt
  State                          Recv-Q                          Send-Q                                                                Local Address:Port                                                       Peer Address:Port
  
  LISTEN                         0                               128                                                             0.0.0.0:8002                                                            0.0.0.0:*
  LISTEN                         0                               128                                                             0.0.0.0:8003                                                            0.0.0.0:*
  LISTEN                         0                               128                                                             0.0.0.0:8004                                                            0.0.0.0:*
  LISTEN                         0                               128                                                             0.0.0.0:8005                                                            0.0.0.0:*
  LISTEN                         0                               128                                                             0.0.0.0:8006                                                            0.0.0.0:*
  LISTEN                         0                               128                                                             0.0.0.0:8007                                                            0.0.0.0:*
  ....
  
  ~# netstat -s | grep -i LISTEN
  2129596 times the listen queue of a socket overflowed
  2138743 SYNs to LISTEN sockets dropped
  </code></pre>
  <p>And every day theses numbers increase.</p>
  <p>Any idea on where the problem is, and how to resolve this ? Thank you!!</p>
  <p><a href="https://i.stack.imgur.com/Nj9ae.png" rel="nofollow noreferrer">enter image description here</a></p>