title: Thousands of IIS requests stuck on EndRequest
body: <p>I've been trying to diagnose an issue pertaining to thousands of hung/stuck <strong>EndRequest</strong> requests in IIS. This is becoming a large problem for us as we're hitting the concurrent connection cap after about a week or two and have to recycle the whole application pool to clear the request list.</p>
  
  <p>Because this is a live application, I have limited troubleshooting options, so anything that would halt or bring down the application pool I am not allowed to do.</p>
  
  <p><strong>IIS Information</strong></p>
  
  <ul>
  <li><p>Concurrent connection cap is set to its maximum of 65535.</p></li>
  <li><p>Configuration debug in the web config is set to false and we have a
  timeout set at 110 seconds.</p></li>
  <li><p>Windows Server 2012 R2 Version 6.2 (Build 9200)</p></li>
  <li><p>IIS Version 8.5.9600.16384</p></li>
  <li><p>The long running requests have 0 data transfer, checked with
  WireShark.</p></li>
  </ul>
  
  <p>I'm pretty much at a loss on why these aren't timing out. I've set all the appropriate settings - the ones I could find from MSDN and other sources. We have a very, very hard time replicating this on our development environment so it's been blind testing for the most part. I've found articles and such on other state hangs, but I cannnot find anything on why a request in the EndRequest state will not time out.</p>
  
  <p>Advanced Settings Page:
  <a href="https://postimg.org/image/gxec32kmt/" rel="nofollow noreferrer">https://postimg.org/image/gxec32kmt/</a></p>
  
  <p>Application Pool Requests Page:
  <a href="https://postimg.org/image/qupcw57o5/" rel="nofollow noreferrer">https://postimg.org/image/qupcw57o5/</a></p>
  
  <p>Web Config:
  <a href="https://postimg.org/image/5xt4rh1xh/" rel="nofollow noreferrer">https://postimg.org/image/5xt4rh1xh/</a></p>
  
  <p><strong>Update 1</strong></p>
  
  <p><s>I did a bit of digging into our fallback that is supposed to close connections after an hour of no usage. We seem to currently have 10,153 sessions still active with a last active time of 3 days ago. I've stepped through this function quite a bit and it seems to be working as intended. It goes through the list of sessions and anyone over an hour of inactivity has their <code>WebSocketHandler.Close()</code> method called. However it seems some sessions are refusing to close after the method is being called. We have logging in place to tell us if any exceptions are being thrown during the run but it seems as though it's running as expected.</s>
  This was my mistake. I was running against an old sessions data pull. A current pull of the session data shows no sessions running greater then their specified time. This means that the <code>WebSocketHandler.Close()</code> was called on them and they were removed from our in-memory list.</p>
  
  <p><strong>Update 2</strong></p>
  
  <p>NETSTAT using netstat -s on pastebin: <a href="https://pastebin.com/embed_js/qBbZ4gJ1" rel="nofollow noreferrer">https://pastebin.com/embed_js/qBbZ4gJ1</a></p>
  
  <p><strong>Update 3</strong></p>
  
  <p>Correction to update 1. Can a connection close be called and fail? If so, then we're accidentally orphaning the reference to the connection in our server. I would still expect the IIS timeout to kick in however, there must be some catches to it collecting requests.</p>