title: Scrapy approach to scraping multiple URLs
body: <p>I have a project which requires a great deal of data scraping to be done.</p>
  
  <p>I've been looking at Scrapy which so far I am very impressed with but I am looking for the best approach to do the following:</p>
  
  <p>1) I want to scrape multiple URL's and pass in the same variable for each URL to be scraped, for example, lets assume I am wanting to return the top result for the keyword "python" from Bing, Google and Yahoo.</p>
  
  <p>I would want to scrape <code>http://www.google.co.uk/q=python</code>, <code>http://www.yahoo.com?q=python</code> and <code>http://www.bing.com/?q=python</code> (not the actual URLs but you get the idea)</p>
  
  <p>I can't find a way to specify dynamic URLs using the keyword, the only option I can think of is to generate a file in PHP or other which builds the URL and specify scrapy to crawl the links in the URL.</p>
  
  <p>2) Obviously each search engine would have its own mark-up so I would need to differentiate between each result to find the corresponding XPath to extract the relevant data from</p>
  
  <p>3) Lastly, I would like to write the results of the scraped Item to a database (probably redis), but only when all 3 URLs have finished scraping, essentially I am wanting to build up a "profile" from the 3 search engines and save the outputted result in one transaction.</p>
  
  <p>If anyone has any thoughts on any of these points I would be very grateful.</p>
  
  <p>Thank you</p>