title: Scraping for research projects: some issues
body: <p>I am scraping some websites for a research project, I am encountering issues that I think might be useful to many SO users.
  Given a well-defined topic (e.g. bird-watching or astro-physics), basically my objectives are:</p>
  
  <ul>
  <li>Identify important websites that spread these ideas</li>
  <li>Crawl a representative sample of these websites</li>
  <li>Perform some network analysis and thematic analysis of the data (e.g. topic models)</li>
  <li>Publish the results in an academic venue, without publishing any of the crawled data</li>
  </ul>
  
  <p>To achieve this goal, I am finding the following obstacles:</p>
  
  <ul>
  <li><strong>Sampling method</strong>: obviously, it's impossible to establish the boundaries of the sites of interest. As it is impossible to know the size of the dataset, how can I establish the representativeness of the sample? I can crawl 10K, 1M, or 10M pages without knowing when I should stop.</li>
  <li><strong>Detection/banning issue</strong>: my crawler, based on <a href="http://scrapy.org" rel="nofollow">Scrapy</a>, is following <code>robots.txt</code> and tries not to hammer servers by introducing increasing delays between the requests, starting from 25ms. However, a lot of servers still detect the crawler and block it. In this sense, my sampling process is totally biased by what servers cut me off.</li>
  <li><strong>Legal issues</strong>: this is clearly a gray area, but I feel that if I don't publish the actual pages I should be on the safe side. What precautions can I take to avoid upsetting people, especially if the results of the study annoy some website owners?</li>
  </ul>
  
  <p>It would be nice to outline a methodology for researchers, because I am sure that many people encounter these problems as soon as they start crawling a non-trivial number of pages.</p>
  
  <p>Thanks for any suggestions!</p>