title: Build vs Buy web scraper , indexer
body: <p>We are just getting started on a  web scraping , and text analytics project.
  I am evaluating build vs Buy</p>
  <p>A few requirements</p>
  <ol>
  <li>Ability to crawl a few 1000s of websites</li>
  <li>Ability to parse html, pdfs, words etc and tokenize them</li>
  <li>Scalable lucene index</li>
  <li>Access to raw content i.e parsed tokenized content for custom analytics</li>
  <li>Of course simple word search TF/IDF</li>
  </ol>
  <p>Are there any existing vendors who can provide these features without charging too much i.e a few 1000s is fine.</p>
  <p>Appreciate any pointers you may have</p>