title: Scrapy - stop requests but process responses
body: <p>I have a Scrapy project with a lot of spiders. There is a server side solution to restart HMA VPN in order to change interface IP(so that we get different IP and don't get blocked).</p>
  
  <p>There is a custom download middleware that sends corresponding socket message for each request and response so that server side solution can trigger VPN restart. Obviously Scrapy must NOT yield any new requests when VPN restart is about to happen - we control that by having a lock file. Scrapy however must handle all not yet received responses before VPN restart can actually happen.</p>
  
  <p>Putting <code>sleep</code> in download middleware stops Scrapy completely. Is there a way to handle responses but hold off new requests(until lock file gets removed)?</p>
  
  <p>This obviously is the case when more then 1x concurrent request is yielded.</p>
  
  <p>Following middleware code is used:</p>
  
  <pre><code>class CustomMiddleware(object):
  
      def process_request(self, request, spider):
          while os.path.exists(LOCK_FILE_PATH):
              time.sleep(10)
  
          # Send corresponding socket message("OPEN")
  
      def process_response(self, request, response, spider):
          # Send corresponding socket message("CLOSE")
          return response
  </code></pre>