title: Is Erlang the right choice for a webcrawler?
body: <p>I am planning to write a webcrawler for a NLP project, that reads in the thread structure of a forum everytime in a specific interval and parses each thread with new content. Via regular expressions, the author, the date and the content of new posts is extracted. The result is then stored in a database.</p>
  
  <p>The language and plattform used for the crawler have to match the following criteria:</p>
  
  <ul>
  <li>easily scalable on multiple cores and cpus</li>
  <li>suited for high I/O loads</li>
  <li>fast regular expression matching</li>
  <li>easily to maintain/few operational overhead</li>
  </ul>
  
  <p>After some research I think Erlang might be a fitting candidate, but I read it's not very good at string processing (and so regular expression matching). Neither do I have any expirience about the maintenance factor. </p>
  
  <p>Is Erlang a good technology for the scenario described above? And if not, what would be a good alternative?</p>