title: Apache proxy is not doing connection re-use with either prefork or worker MPM
body: <p><strong>Background Info</strong></p>
  
  <p>We have a Dell R620 telecom based enterprise application server runs on RHEL 5 platform &amp; deployed with both Apache web server &amp; OCCAS application server. We have proprietary phones which connects to the application server as clients &amp; hits the Apache for all HTTP based requests. The Apache server is configured to proxy/forward the requests to OCCAS.</p>
  
  <p><strong>The Problem</strong></p>
  
  <p>The proprietary phones will send around 350 HTTP requests (during start-up) within 35 seconds i.e., around 10 requests per second. We could clearly see the connection between the phone &amp; Apache is reusing the connections, whereas, the connections between Apache i.e., mod_proxy &amp; OCCAS <strong>is NOT reusing connection</strong>, ending up creating one connection per request. The connection between Apache &amp; OCCAS is closed by Apache/proxy (as we see FIN from proxy) right after the request is serviced. So, we could see 350 sockets in TIME_WAIT state for a minute after all 350 requests are serviced. Our system is supposed to handle 500 such phones. Now, if you can imagine when all 500 phones making 350 requests pretty much at the same time, it would require 175,000 (350 * 500) ephemeral ports! Hence, I was trying to make connection reuse to work between Apache proxy &amp; OCCAS, for which I'm facing issues.</p>
  
  <p>I have posted below the configurations we have &amp; different things we tried &amp; failed to make connection reuse to work.</p>
  
  <p><strong>Our configurations</strong></p>
  
  <p>1) The prefork &amp; worker MPM configured in our system in /etc/httpd/conf/httpd.conf is as follows &amp; we are using the default prefork MPM.</p>
  
  <pre><code>&lt;IfModule prefork.c&gt;
  StartServers       8
  MinSpareServers    5
  MaxSpareServers   20
  ServerLimit      256
  MaxClients       256
  MaxRequestsPerChild  4000
  &lt;/IfModule&gt;
  
  &lt;IfModule worker.c&gt;
  StartServers         8
  MaxClients         150
  MinSpareThreads      5
  MaxSpareThreads     20
  ThreadsPerChild    256
  MaxRequestsPerChild  4000
  &lt;/IfModule&gt;
  </code></pre>
  
  <p>2) Keepalive related parameters in /etc/httpd/conf/httpd.conf is as follows:</p>
  
  <pre><code>[root@host ~]# grep -i keepalive /etc/httpd/conf/httpd.conf | grep -v "#"
  KeepAlive On
  MaxKeepAliveRequests 100
  KeepAliveTimeout 15
  BrowserMatch "Mozilla/2" nokeepalive
  BrowserMatch "MSIE 4\.0b2;" nokeepalive downgrade-1.0 force-response-1.0
  [root@host ~]#
  </code></pre>
  
  <p>3) The proxy configuration is as follows:</p>
  
  <pre><code>&lt;VirtualHost *:80&gt;
  
  &lt;Proxy *&gt;
      Order deny,allow
      Allow from all
  &lt;/Proxy&gt;
  
  &lt;Location /ds-webservice&gt;
  ProxyPass         http://{IP address of local host}:8001/ds-webservice retry=0
  ProxyPassReverse  http://{IP address of local host}:8001/ds-webservice
  &lt;/Location&gt;
  .
  .
  # Few more similar ProxyPass
  .
  .
  &lt;/VirtualHost&gt;
  </code></pre>
  
  <p>4) We use default prefork Apache:</p>
  
  <pre><code>[root@host ~]# httpd -l
  Compiled in modules:
    core.c
    prefork.c
    http_core.c
    mod_so.c
  </code></pre>
  
  <p><strong>Tests tried but failed to achieve connection reuse</strong></p>
  
  <p><em>Test 1)</em> We noticed the mod_proxy is sending 'Connection: Keep-alive' header (but NOT 'Keep-alive' header itself) in the forwarded requests whereas OCCAS does NOT respond with 'Connection' or 'Keep-alive' headers. Our initial suspect was Apache does not like the response in absence of these headers. So, as an experiment, we forwarded the requests to another Apache in which we did get back 'Connection' or 'Keep-alive' headers in response from other Apache to mod_proxy, and still the mod_proxy did NOT do connection reuse. So, we confirmed its the mod_proxy which has the issue (or may be some misconfiguration from our side).</p>
  
  <p><em>Test 2)</em> Researching a bit, we found a bug report on connection reuse with prefork in 2.2.0 - <a href="https://issues.apache.org/bugzilla/show_bug.cgi?id=38602" rel="nofollow noreferrer">https://issues.apache.org/bugzilla/show_bug.cgi?id=38602</a>. The bug 38602 is fixed in 2.2.1. However, based on the post <a href="https://stackoverflow.com/questions/9221640/can-i-use-apache-mod-proxy-as-a-connection-pool-under-the-prefork-mpm">Can I use Apache mod_proxy as a connection pool, under the Prefork MPM?</a> we found the issue seem to persist even after upgrading. So, just to be sure, we upgraded our Apache from 2.2.3 to 2.2.26 but mod_proxy did NOT do connection reuse.</p>
  
  <p><em>Test 3)</em> We tried few tests with with different ProxyPass parameters (without changing anything in httpd.conf) &amp; none of them helped i.e., mod_proxy did NOT do connection reuse:
  NOTE: Excluding (7), we added the below parameters next to 'ProxyPass http://{IP address of local host}:8001/ds-webservice' for our tests.<br>
   1. retry=0 <em>(Our default configuration)</em><br>
   2. disablereuse=OFF<br>
   3. max=256 retry=0<br>
   4. proxyto=80<br>
   5. retry=0 keepalive=ON<br>
   6. retry=300 smax=5 max=20 ttl=120<br>
   7. And lastly, we added keep-alive specific parameters inside the proxy configuration hoping it'll override the keep-alive parameters in http.conf :  </p>
  
  <pre><code>&lt;VirtualHost *:80&gt;
  
  Timeout 300
  KeepAlive On
  MaxKeepAliveRequests 1000
  KeepAliveTimeout 100
  ProxyRequests Off
  
  &lt;Proxy *&gt;
      Order deny,allow
      Allow from all
  &lt;/Proxy&gt;
  
  &lt;Location /ds-webservice&gt;
  ProxyPass         http://{IP address of local host}:8001/ds-webservice retry=0
  ProxyPassReverse  http://{IP address of local host}:8001/ds-webservice
  &lt;/Location&gt;
  .
  .
  # Few more similar ProxyPass
  .
  .
  &lt;/VirtualHost&gt;
  </code></pre>
  
  <p><em>Test 4)</em> Based on the post <a href="https://stackoverflow.com/questions/9221640/can-i-use-apache-mod-proxy-as-a-connection-pool-under-the-prefork-mpm">Can I use Apache mod_proxy as a connection pool, under the Prefork MPM?</a>, we switched to prefork Apache to worker Apache by using httpd.worker on 2.2.26 &amp; mod_proxy did NOT do connection reuse.</p>
  
  <p>It seems we are missing something very fundamental here. Can you guys point me if there is anything we did wrong either in ProxyPass or prefork configuration as both are quite related? Any pointers will be of great help.</p>
  
  <p>Thanking you in advance!</p>
  
  <p>Regards<br>
  Goutham Prasad</p>