title: Web page crawling in C#
body: <p>I have been given a task to crawl / parse and index available books on many library web page. I usually use HTML Agility Pack and C# to parse web site content.  One of them is the following:</p>
  
  <p><a href="http://bibliotek.kristianstad.se/pls/bookit/pkg_www_misc.print_index?in_language_id=en_GB" rel="nofollow">http://bibliotek.kristianstad.se/pls/bookit/pkg_www_misc.print_index?in_language_id=en_GB</a></p>
  
  <p>If you search for a * (all books) it will return many lists of books, paginated by 10 books per page. </p>
  
  <p>Typical web crawlers that I have found fail on this website. I have also tried to write my own crawler, which would go through all links on the page and generate post/get variables to dynamically generate results. I havent been able to do this as well, mostly due to some 404 errors that I get (although I am certain that the links generated are correct).</p>
  
  <p>The site relies on javascript to generate content, and uses a mixed mode of GET and POST variable submission. </p>