title: Latency Between Apache And Tomcat
body: <p>We have a setup where we have a Elastic LB that spreads load over two Apache servers A1 and A2. These apache servers render a few php pages and primarily redirects the API requests to tomcat appplication servers T1 and T2 as per the following diagram:</p>
  
  <pre><code>Incoming Request
      |
      |
      |
      \/
      LB
      /\
     /  \
    /    \
  A1     A2
  |\     /|
  | \   / |
  |  \ /  |
  |  / \  |
  T1      T2
  </code></pre>
  
  <p>We have recently started to note delays between apache and tomcat. Here is an example log lines from apache mod_slow log and tomcat access log for the same request:</p>
  
  <pre><code>APACHE_MOD_SLOW: VNSdtwoAAJkAACnXb-cAAACJ [06/Feb/2015:16:25:51 +0530] elapsed: 50.58 cpu: 0.00(usr)/0.00(sys) pid: 10711 ip: 10.0.0.153 host: www.example.com:443 reqinfo: GET /data/v1/url?url=test-508324 HTTP/1.1
  
  TOMCAT: [06/Feb/2015:16:26:42 +0530] "GET /data/v1/url?url=test-508324 HTTP/1.1" 200 65 10
  </code></pre>
  
  <p>Apache says the incoming request came at <code>06/Feb/2015:16:25:51 +0530</code> and it took <code>50s</code> to process the request. Whereas tomcat says it took only <code>10ms</code> to process the request whereas it received the request at <code>06/Feb/2015:16:26:42 +0530</code>. </p>
  
  <p>It means it took nearly 50s for apache to connect and send the whole request to tomcat. Apache is using <code>mod_proxy_ajp</code> to connect to apache.   Here is the configuration:</p>
  
  <pre><code>&lt;Proxy balancer://prod&gt;
     BalancerMember ajp://127.0.0.1:8009 route=jvmRoute-8009 connectiontimeout=1 retry=300
     BalancerMember ajp://10.0.0.153:8009 route=jvmRoute-8009 connectiontimeout=1 retry=300    
     ProxySet lbmethod=byrequests
  &lt;/Proxy&gt;
  </code></pre>
  
  <p>Here is the connector configuration from tomcat:</p>
  
  <pre><code>&lt;Connector port="8009" protocol="AJP/1.3" redirectPort="8443" maxThreads="4096" minSpareThreads="25" maxSpareThreads="75"/&gt;
  </code></pre>
  
  <p>As per connectiontimeout value, I am assuming apache shouldn't take more than 1 sec to establish connection. Since both apache and tomcat are both on the same machine, there shouldn't be much time lag once connection is established.</p>
  
  <p>If it helps, we are using <code>https</code> requests. But that, I don'y think has anything to do with this. We have done <code>ab</code> test in order to compare performance using <code>https</code>, <code>http</code> and connecting tomcat directly. Here are the stats:</p>
  
  <pre><code>ab -n5000 -c5 https://example.com/test/100001                                        
  
  Requests per second:    13.67 [#/sec] (mean)                                        
  Time per request:       365.851 [ms] (mean)                                        
  Time per request:       73.170 [ms] (mean, across all concurrent requests)                                        
  Transfer rate:          79.96 [Kbytes/sec] received                                        
  
  Connection Times (ms)                                        
                min  mean[+/-sd] median   max                                        
  Connect:      236  267  95.5    247    3401                                        
  Processing:    83   98  58.6     89    1959                                        
  Waiting:       82   96  57.5     87    1959                                        
  Total:        319  365 134.0    338    3571                                        
  
  Percentage of the requests served within a certain time (ms)                                        
    50%    338                                        
    66%    347                                        
    75%    356                                        
    80%    364                                        
    90%    399                                        
    95%    477                                        
    98%    689                                        
    99%    869                                        
   100%   3571 (longest request)
  
  ab -n5000 -c5 http://example.com/test/100001                                        
  
  Time per request:       186.015 [ms] (mean)
  Time per request:       37.203 [ms] (mean, across all concurrent requests)
  Transfer rate:          155.55 [Kbytes/sec] received
  
  Connection Times (ms)
                min  mean[+/-sd] median   max
  Connect:       74   79  33.4     76    1278
  Processing:    83  107  82.3     91    3964
  Waiting:       82  105  60.9     89     940
  Total:        157  186  90.1    168    4042
  
  Percentage of the requests served within a certain time (ms)
    50%    168
    66%    174
    75%    180
    80%    184
    90%    211
    95%    259
    98%    379
    99%    507
   100%   4042 (longest request)
  
  ab -n5000 -c5 http://IP:8080/test/100001                                        
  
  Requests per second:    31.32 [#/sec] (mean)
  Time per request:       159.624 [ms] (mean)
  Time per request:       31.925 [ms] (mean, across all concurrent requests)
  Transfer rate:          181.30 [Kbytes/sec] received
  
  Connection Times (ms)
                min  mean[+/-sd] median   max
  Connect:       71   76  68.4     73    3079
  Processing:    78   84  13.1     81     594
  Waiting:       77   83   6.5     81     185
  Total:        149  159  71.2    154    3313
  
  Percentage of the requests served within a certain time (ms)
    50%    154
    66%    157
    75%    160
    80%    161
    90%    166
    95%    171
    98%    177
    99%    189
   100%   3313 (longest request)
  </code></pre>
  
  <p>Following observation leads me to believe this that bad performance is depending on sequence of events because none of the requests in the test performed that badly.</p>
  
  <p>Versions:
      Apache: 2.2.4
      Tomcat: 8.0.16</p>
  
  <p>Any ideas where the lag is coming from and how to reduce the  same?</p>