title: Optimize and Scale my Webservers (Nginx + Gunicorn + Django stack)
body: <p>We have a web server setup running Nginx and Gunicorn which runs a Django application. There have been times that we received high load and got a lot of 5XXs. To get rid of that immediately we increased the number of servers 5 times and restarted all Gunicorn workers. 
  Now, our servers are still not used optimally, CPU is around 10-15% max used and same for the memory (20-30%). 
  So, this is not a permanent solution and it also incurs us downtime and high costs if we go with larger and more number of servers. </p>
  
  <p>To research this we started with <strong>load tests.</strong> </p>
  
  <p>When we do a load test we start getting 5xx errors even when the resources on the server are barely used.</p>
  
  <h1><strong>LOAD TEST RESULTS</strong></h1>
  
  <p><strong>Load Test 1</strong></p>
  
  <p><strong>Resources</strong>: 1 ELB 1 server with Nginx and 4 workers of gunicorn running in virtual env serving Django. </p>
  
  <p>Ran load test using Locust which has a full workflow of what a user would do after getting into the app.</p>
  
  <p>1000 Users, Ramp up 10 </p>
  
  <p><strong>Requests Per Second</strong> (Before Failure): 120
  <strong>Max CPU</strong> = 45 % on CloudWatch, on server fluctuating up to 100%
  Memory did not get used more than 2 GB after load test started. Server has 8 GB Memory (RAM) </p>
  
  <p>Suddenly I start getting errors. Started getting gateway timeouts, cannot connect to backend server. There were AWS ELB 5xx as well and then the server also went out of service.</p>
  
  <p><strong>Load Test 2</strong> </p>
  
  <p><strong>Resources</strong>: 4 servers, 1 ELB, with Nginx and 4 workers of Gunicorn running on each server in virtualenv serving Django.</p>
  
  <p>1000 Users, Ramp up 10 </p>
  
  <p>34 RPS, 11 % Failure rate suddenly and then stopped getting 200. 
  ELB servers out of service. Around 29000 total requests, 2965 had failed I believe and then all started failing. The size of requests around 140688 Bytes.Started getting gateway timeouts, cannot connect to the backend server.</p>
  
  <p>CPU again did not go above 20 % utilization.
  Memory did not get used more than 2 GB after load test started. Server has 8 GB Memory (RAM) </p>
  
  <p>Load Test 3
  <strong>Resources</strong>: 11 servers under one ELB</p>
  
  <p>Went till Fluctuated from 80-200 RPS (requests per second)
  Started getting gateway timeouts, cannot connect to backend server
  Servers got unhealthy</p>
  
  <p>CPU again did not go above 20 % utilization.
  Memory did not get used more than 2 GB after load test started. Server has 8 GB Memory (RAM) </p>
  
  <p><strong>Load Test 4</strong></p>
  
  <p>Resources: 40 servers
  Results on similar lines.</p>
  
  <h2>Tweaking Starts</h2>
  
  <h2><strong>Things I have configured on Nginx</strong></h2>
  
  <p>Allowed to accept 1500 connections per worker and fine tuned fds, cache, gzip, worker limit etc. </p>
  
  <p><strong>Made the following changes to Nginx conf:</strong></p>
  
  <pre><code>Nginx worker processes are set to auto which is okay, I guess.
  Nginx keepalive timeout is 65 which is high. 
  
  open_file_cache max=200000 inactive=20s;
  open_file_cache_valid 30s;
  open_file_cache_min_uses 2;
  open_file_cache_errors on;
  
  tcp_nopush=on     Send all the response in a simple packet.
  
  keepalive_timeout 65;
  types_hash_max_size 2048;
  send_timeout 10;
  sendfile on;    sendfile_max_chunk 512k;
  worker_rlimit_nofile 10000;
  </code></pre>
  
  <p><strong>Gzip settings as follows:</strong></p>
  
  <pre><code>gzip on;
      gzip_disable "msie6";
  
      gzip_vary on;
      gzip_proxied any;
      gzip_comp_level 6;
      gzip_buffers 16 8k;
      gzip_http_version 1.1;
      gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
  </code></pre>
  
  <p>Could it be that my Nginx is accepting connections and Gunicorn not being optimized is not being able to process those requests?</p>
  
  <h2><strong>System Level</strong></h2>
  
  <p>Also, should I make system level changed like:
  sysctl net.core.somaxconn=1024 ? For now, it is 128. Is that the limiting problem?</p>
  
  <p><strong>Gunicorn Command we are using</strong></p>
  
  <pre><code>exec gunicorn ${DJANGO_WSGI_MODULE}:application \
    -b 127.0.0.1:8000 \
    --name $NAME \
    --workers $NUM_WORKERS \
    --user=$USER --group=$GROUP \
    --log-level=info \
    --bind=unix:$SOCKFILE \
    --timeout $TIMEOUT \
    --backlog=2048
  </code></pre>
  
  <p>Workers on m4.large is 4. Timeout and other variable values:</p>
  
  <pre><code>SOCKFILE=/home/ubuntu/run/gunicorn.sock    # using this unix socket
  USER=ubuntu                                   # the user to run as
  GROUP=ubuntu                                  # the group to run as
  NUM_WORKERS=4                              # how many workerprocesses should Gunicorn spawn
  DJANGO_WSGI_MODULE=haptik_api.wsgi               # WSGI module name
  PORT=8000
  TIMEOUT=300
  </code></pre>
  
  <h1>FINAL PROBLEMS</h1>
  
  <ol>
  <li>I want to know how and what parameters I can change in Gunicorn so that I am well utilizing my resources. My servers are 4 core 8 GB RAM for now. </li>
  <li>How can I make Gunicorn use more memory and CPU so that the servers are use optimally. What all can I tweak?</li>
  <li>Is there system level tweak required? like sysctl</li>
  <li>I have even tried with m4.xlarge and m4.2xlarge and t2.medium (with 4 times the count of servers)</li>
  <li>Can AWS ELB be the problem too? That it can't handle those requests and can't add to surge queue?</li>
  <li>How do I optimize my Nginx + Gunicorn + Django nonrel to accept more connections and perform better and also optimize resource utilization?</li>
  <li>What should I set as SCALING POLICY for the servers? CPU? Memory? Network? or anything else. (Problem also is they don't change much when the load increases). Servers are on AWS - AWS EC2s. </li>
  </ol>
  
  <p>Please let me know if you need more details. </p>