title: High number of TIME_WAITs in Haproxy
body: <p>We have haproxy 1.3.26 hosted on CentOS 5.9 machine having 2.13 GHz Intel Xeon processor which is acting as a http &amp; tcp load balancer for numerous services, serving a peak throughput of ~2000 requests/second. It has been running fine for 2 years but gradually both traffic and number of services are increasing.</p>
  
  <p>Off late we've observed that even after reload old haproxy process remains. On further investigation we found that old process has numerous connections in TIME_WAIT state. We also saw that <code>netstat</code> and <code>lsof</code> were taking a long long time. On referring <a href="http://agiletesting.blogspot.in/2013/07/the-mystery-of-stale-haproxy-processes.html" rel="nofollow noreferrer">http://agiletesting.blogspot.in/2013/07/the-mystery-of-stale-haproxy-processes.html</a> we introduced <code>option forceclose</code> but it was messing up with various monitoring service hence reverted it. On further digging we realised that in <code>/proc/net/sockstat</code> close to 200K sockets are in <code>tw</code> (<code>TIME_WAIT</code>) state which is surprising as in <code>/etc/haproxy/haproxy.cfg</code> <code>maxconn</code> has been specified as 31000 and <code>ulimit-n</code> as 64000. We had <code>timeout server</code> and <code>timeout client</code> as <code>300s</code> which we changed to <code>30s</code> but not much use. </p>
  
  <p>Now the doubts are :-</p>
  
  <ul>
  <li>Whether such a high number of TIME_WAITs is acceptable. If yes whats a number after which we should be worried. Looking at <a href="https://stackoverflow.com/questions/1803566/what-is-the-cost-of-many-time-wait-on-the-server-side">What is the cost of many TIME_WAIT on the server side?</a> and <a href="https://stackoverflow.com/questions/337115/setting-time-wait-tcp">Setting TIME_WAIT TCP</a> seems there shouldn't be any issue.</li>
  <li>How to decrease these TIME_WAITs</li>
  <li>Are there any alternatives to netstat and lsof which will perform fine even if there are very high number of TIME_WAITs</li>
  </ul>