title: Free Large datasets to experiment with Hadoop
body: <p>Do you know any large dataset to experiment with Hadoop which is free/low cost?
  Any pointers/links related are appreciated.</p>
  
  <p>Preference:</p>
  
  <ul>
  <li><p>At least one GB of data.</p></li>
  <li><p>Production log data of webserver.</p></li>
  </ul>
  
  <p>Few of them which I found so far:</p>
  
  <ol>
  <li><p><a href="http://dumps.wikimedia.org/enwiki/" rel="nofollow noreferrer">Wikipedia dump</a></p></li>
  <li><p><a href="http://wiki.freebase.com/wiki/Data_dumps" rel="nofollow noreferrer">http://wiki.freebase.com/wiki/Data_dumps</a></p></li>
  <li><p><a href="http://aws.amazon.com/publicdatasets/" rel="nofollow noreferrer">http://aws.amazon.com/publicdatasets/</a></p></li>
  </ol>
  
  <p>Also can we run our own crawler to gather data from sites e.g. Wikipedia? Any pointers on how to do this is appreciated as well.</p>