title: How to make asynchronous HTTP GET requests in Python and pass response object to a function
body: <p>Update: <a href="https://stackoverflow.com/a/15594097/1623879">Problem was incomplete documentation, event dispatcher passing kwargs to the hook function.</a></p>
  
  <p>I have a list of about 30k URLs that I want to check for various strings. I have a working version of this script using Requests &amp; BeautifulSoup, but it doesn't use threading or asynchronous requests so it's incredibly slow.</p>
  
  <p>Ultimately what I would like to do is cache the html for each URL so I can run multiple checks without making redundant HTTP requests to each site. If I have a function that will store the html, what's the best way to asynchronously send the HTTP GET requests and then pass the response objects?</p>
  
  <p>I've been trying to use Grequests (<a href="https://stackoverflow.com/questions/10555292/ideal-method-for-sending-multiple-http-requests-over-python">as described here</a>) and the "hooks" parameter, but I'm getting errors and the <a href="http://docs.python-requests.org/en/latest/user/advanced/#event-hooks" rel="nofollow noreferrer">documentation doesn't go very in-depth</a>. So I'm hoping someone with more experience can shed some light.</p>
  
  <p>Here's a simplified example of what I'm trying to accomplish:</p>
  
  <pre><code>import grequests
  
  urls = ['http://www.google.com/finance','http://finance.yahoo.com/','http://www.bloomberg.com/']
  
  def print_url(r):
      print r.url
  
  def async(url_list):
      sites = []
      for u in url_list:
          rs = grequests.get(u, hooks=dict(response=print_url))
          sites.append(rs)
      return grequests.map(sites)
  
  print async(urls)
  </code></pre>
  
  <p>And it produces the following TypeError:</p>
  
  <pre><code>TypeError: print_url() got an unexpected keyword argument 'verify'
  &lt;Greenlet at 0x32803d8L: &lt;bound method AsyncRequest.send of &lt;grequests.AsyncRequest object at 0x00000000028D2160&gt;&gt;
  (stream=False)&gt; failed with TypeError
  </code></pre>
  
  <p>Not sure why it's sending 'verify' as a keyword argument by default; it would be great to get something working though, so if anyone has any suggestions (using grequests or otherwise) please share :)</p>
  
  <p>Thanks in advance.</p>